US 20020087601A1 

(19) United States 

(12) Patent Application Publication (io) Pub. No.: US 2002/0087601 Al 

Anderson et al. (43) Pub. Date: Jul. 4, 2002 



(54) CLIENT-SERVER SYSTEM FOR MERGING 
OF METADATA WITH IMAGES 

(76) Inventors: Eric C. Anderson, San Jose, CA (US); 

John Bernstein, San Jose, CA (US); 
Robert Paul Morris, Raleigh, NC (US) 

Correspondence Address: 
Mr. Stephen G. Sullivan 
SAWYER LAW GROUP LLP 
P.O. Box 51418 
Palo AIto„ CA 94303 (US) 

(21) Appl. No.: 09/796,179 

(22) Filed: Feb. 27, 2001 

Related U.S. Application Data 

(63) Continuation-in-part of application No. 09/752,082, 
filed on Dec. 29, 2000. 



Publication Classification 

(51) Int. CI. 7 G06F 17/24 

(52) U.S. CI 707/515; 707/530 

(57) ABSTRACT 

A method and system for merging metadata with digital 
images is disclosed, where the digital images are stored as 
image files having metadata fields associated therewith. The 
method and system include receiving data uploaded from a 
remote location, providing a web application to a client 
device over a network when requested, and transmitting the 
previously uploaded data to the web application. When an 
image requires input of metadata on the client device, the 
uploaded data is displayed on the client device for selection 
by the user. In response to the user's selection, the selected 
data is then used to automatically populate the metadata 
fields associated with the image file. 
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CLIENT-SERVER SYSTEM FOR MERGING OF 
METADATA WITH IMAGES 

CROSS-REFERENCE TO RELATED 
APPLICATION 

[0001] The present invention is continuation-in-part of 
co-pending U.S. patent application Ser. No. 09/752,082, 
entitled "Meta- Application Architecture For Integrating 
Photo-Service Websites For Browser-Enabled Devices," 
(P212) filed on Dec. 29, 2000, and assigned to the assignee 
of the present invention, and hereby incorporated by refer- 
ence. 

FIELD OF INVENTION 

[0002] The present invention relates to manipulating digi- 
tal images over the Internet, and more particularly to pro- 
viding an architecture for associating metadata with digital 
images on client devices. 

BACKGROUND 

[0003] One advantage of digital cameras over film based 
cameras is that digital cameras have the ability to associate 
metadata with a captured image that may later be accessed 
by a viewer. For example, once the digital image has been 
transferred from the digital camera to a PC and displayed, a 
user may also view information about the image, such as 
date and time of capture, the name of the image, and image 
size, for instance. Metadata may be either stored with the 
image or stored separately in a separate database. If the 
metadata is stored with the image, then the metadata is 
typically stored as tags within the image file of the captured 
image. 

[0004] For example, a standards organization referred to 
as the Digital Imaging Group (DIG) has developed a DIG35 
Metadata Specification that provides a set of metadata 
definitions to the imaging industry. The DIG35 metadata, 
specification for describing elements of an image is inde- 
pendent of the imaging application or the image file format 
used. 

[0005] By associating metadata with digital images, every 
image that is uploaded to the Internet and shared with others 
around the globe may be its own stand-alone catalog record 
that can be used by each recipient for a variety of purposes. 

[0006] Businesses, professionals and consumers can all 
utilize metadata in order to manage images. One example 
use of metadata is professional photographers associating 
information about camera settings, copyright information 
and image manipulation techniques within the image in 
order to recreate images and document their work. Addi- 
tionally, metadata enables consumers to share their captured 
experiences by using metadata to tell the story or narration 
behind their images. For example, users may use image- 
editing software on their PC to add explanatory captions to 
each photo from their vacation, and then upload those photos 
to a photo-sharing website. Assuming the photo website was 
DIG35 compatible, the photo website would understand and 
save all the captions. Friends and family may then access the 
photo website and not only see the pictures, but read and/or 
hear the story and history behind each image by accessing 
the metadata. 

[0007] The assignee of the present application has devel- 
oped an operating system for digital cameras that generates 



digital image files that include metadata. Some of the 
metadata is populated automatically by the camera upon 
image capture with data describing various attributes of the 
image and the camera. Other metadata may be manually 
entered by the user, such as image names, categories and 
captions. Still other metadata may be populated by software 
programs that run on the camera. 

[0008] Although these methods effectively populate the 
metadata, they have disadvantages. The disadvantage with 
populating the metadata manually is that the user must 
retype the information for each image. When providing 
information for large numbers of images, manually entering 
the data for each image is extremely tedious and time- 
consuming for the user, especially when the images share 
many of the same characteristics. This problem can be 
alleviated by running a program on the camera that auto- 
matically populates the metadata with the desired data. 
However, if the user uses a different camera and doesn't 
have the program handy, the user must resort to the manual 
method. 

[0009] Accordingly, what is needed is an improved 
method and system for merging metadata with images. The 
present invention addresses such a need. 

SUMMARY OF TOE INVENTION 

[0010] The present invention provides a method and sys- 
tem for merging metadata with digital images, where the 
digital images are stored as image files having metadata 
fields associated therewith. The method and system include 
receiving data uploaded from a remote location to a web 
server, providing a web application to a client device over a 
network when requested, and transmitting the previously 
uploaded data to the web application. When an image 
requires input of metadata on the client device, the uploaded 
data is displayed on the client device for selection by the 
user. In response to the user's selection, the selected data is 
then used to automatically populate the metadata fields 
associated with the image file. 

[0011] According to the present invention, the need for 
cumbersome data entry on the client device is significantly 
minimized, especially where the user needs to supply meta- 
data for a series of images that have similar content. 

DESCRIPTION OF THE DRAWINGS 

[0012] FIG. 1 is a block diagram illustrating a meta- 
application architecture for an online system in accordance 
with a preferred embodiment of the present invention. 

[0013] FIG. 2 showing example metadata elements asso- 
ciated with images for the corporate directory application, 
the user would then provide for each photo the appropriate 
information to populate the fields corresponding to each of 
the metadata elements. 

[0014] FIG. 3 is a flow chart illustrating a process for 
merging metadata with digital images in accordance with 
one preferred embodiment of the present invention. 

[0015] FIG. 4 is a diagram showing the data that a user 
may upload in a corporate directory example. 

[0016] FIG. 5 is a block diagram illustrating two addi- 
tional data sets that have been uploaded and associated with 
John Smith's account. 
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[0017] FIG. 6 is a block diagram illustrating that the 
Employee database that is associated with John Smith's 
account could be further associated with a "Corporate Direc- 
tory" web application. 

[0018] FIG. 7 is a block diagram illustrating a history list 
associated with the user's account and a web application. 

DETAILED DESCRIPTION OF THE 
INVENTION 

[0019] The present invention relates to an online digital 
imaging architecture. The following description is presented 
to enable one of ordinary skill in the art to make and use the 
invention and is provided in the context of a patent appli- 
cation and its requirements. Various modifications to the 
preferred embodiment will be readily apparent to those 
skilled in the art and the generic principles herein may be 
applied to other embodiments. Thus, the present invention is 
not intended to be limited to the embodiments shown but is 
to be accorded the widest scope consistent with the prin- 
ciples and features described herein. 

[0020] The present invention provides a meta-application 
architecture for allowing photo-service websites to receive 
and send images to and from a wide range of client device 
types, and for integrating the services of the photo-service 
sites 14 for access by users of the client devices. The present 
invention also allows client devices to access a variety of 
web applications and to automatically allow the web appli- 
cations to populate the image files on the client devices with 
metadata. 

[0021] FIG. 1 is a block diagram illustrating a meta- 
application architecture for an online system 10 in accor- 
dance with a preferred embodiment of the present invention. 
The system 10 includes multiple client devices 12 that 
request imaging services from multiple online photo-service 
sites 14. The photo -service sites 14 are sites on the Internet 
that provide different types of digital imaging services. For 
example, one photo -service site 14 may provide an image 
hosting service, while another photo-service site 14 provides 
image printing services, for instance. 

[0022] A client device 12 refers to an electronic device 
capable of capturing and/or displaying digital images and 
communicating over a network, such as the Internet. Such 
electronic devices include devices that store digital images, 
such as PCs and photo kiosks, and image capture devices 
such as digital cameras and PDAs, and cellphones that have 
lens attachments, for example. In a preferred embodiment, 
the client devices 12 are browser-based, although non- 
browser-based devices may also be supported. The client 
devices 12 communicate over the Internet via a wireless, or 
wired connection, but because they incorporate different 
browser types, client devices 12 communicate data in dif- 
ferent formats. For example, some client devices 12 such as 
PCs may communicate data in HTML format. Other client 
devices 12 such as cellphones, may use data formats such as 
Wireless Markup Language (WML), which is a streamlined 
version of HTML for small screen displays, cHTML which 
is a subset of HTML, and Handheld Device Markup Lan- 
guage (HDML), for instance. 

[0023] In one embodiment, the client devices 12 connect 
to the Internet via a service provider 16, which may include 
a wireless carrier and/or an Internet service provider (ISP). 



Once connected to the Internet, the client devices 12 have 
the capability of uploading the digital images to the online 
photo-service sites 14 for storage and/or for receiving digital , 
images from the photo-service sites 14 for display. 

[0024] One aspect of the present invention provides a 
meta-application 22 architecture that provides a common 
communication framework for integrating photo -service 
sites 14 and services for client devices 12. The meta- 
application architecture includes a site on the Internet, 
referred to as the image gateway 18, that interfaces between 
the client devices 12 and the photo-service sites 14. In a 
preferred embodiment, the image gateway 18 includes a 
gateway server 20, a software meta-application 22, and a set 
of site adapter software 24 that provide a set of standard 
APIs and data formats that the photo-service sites 14 use so 
that the image gateway 18 can present data and services 
from the sites 14 to the various client devices 12. These same 
APIs and data formats allow the image gateway 18 to 
present the services of multiple photo-services 14 in one 
integrated application, and allow communication among the 
photo-services sites 14. For example, the image gateway 18 
enables a user with images stored on one photo-host site to 
access to the services of all print service providers who also 
use the image gateway 18. The photo-hosting site 14 would 
not need to make any special effort in order to work with the 
print service providers since they are all bound together by 
the meta-application. 

[0025] The gateway server 20 performs the function of a 
web server, while the meta-application 22 performs the 
function of an application server. The meta-application 22 
may reside on the same or different computer from the 
gateway server 20, and one of the photo-service sites may be 
part of the image gateway 18. The gateway server 20 
provides client device 12 connectivity and is primarily 
responsible for detecting the client device 12 type and its 
browser type and display characteristics. The gateway server 
20 may also provide security, configuration, and adminis- 
tration services, including the collection of usage statistics. 

[0026] In a preferred embodiment, the gateway server 20 
passes the data and service requests of the client devices 12 
to and from the meta-application 22 in a device independent 
fashion. 

[0027] One reason that there is no standard for commu- 
nication between photoservice sites 14 is because each 
photo-service site 14 represents its own data and services in 
different formats. For example, all photo-hosting web sites 
14 organize a user's images in a nested tree-like structure 
similar to a file directory, but the names of the nodes in these 
trees vary across sites. For instance, some of the terms used 
include "album/'"pholio,""page," and "shelves". 

[0028] According to another aspect of the present inven- 
tion, the meta-application 22 abstracts the underlying data 
model and the function provided by the photo-service sites 
14, which is common across the photo-service sites 14, to 
define a common data model format for the data, referred to 
here as a meta photo-service model. In a preferred embodi- 
ment, the meta photo-service model is implemented using 
XML. 

[0029] Since each photo-service site 14 may use its own 
data model and define its own API or protocol for accessing 
the site's functionality, respective site adapters 24 are used 
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to convert between the data and service formats of each 
photo-service site 14 and the meta photo-service model 26. 

[0030] In a preferred embodiment, the image gateway 18 
is provided with a database 32 for supporting the aggrega- 
tion of data and services across the various photo-service 
sites 14. This enables the image gateway 18 to support a 
single login for a particular client device 12 and enables data 
sharing, such as billing information, across photo -service 
sites 14. This data sharing eliminates the need for users to 
reenter this information for each site, but requires that the 
database 32 be synchronized with the data stored on the 
photo serving sites. 

[0031] According to a farther aspect of the present inven- 
tion, developers who have registered with the image gate- 
way 18 may post web applications 42 on the image gateway 
18 for access by the client devices 12. In a preferred 
embodiment, the web applications 42 are imaging related 
and allow the users of the client devices 12 to manipulate 
their images and metadata associated with those images. 
Examples of such imagjng-related web applications 42 that 
may be provided include an upload image application that 
uploads images from the client device 12 to a photo-hosting 
service 14 via the image gateway 18, and a search applica- 
tion that searches for the user's images, for instance. The 
web applications 42 may reside on servers external to the 
image gateway 18 or they may reside within the image 
gateway 18. 

[0032] It should be noted that although the term images as 
used herein includes media types such as still images, burst 
images; and time lapse images, the term images also encom- 
passes media types such as movies, sound annotations, 
animations, and clip art, for instance. 

[0033] In a preferred environment, the web applications 
42 are implemented as server-side processes that allow web 
pages to interact with databases and other applications. 
Examples of such server-side processes include active server 
pages (ASPs), CGI scripts and JavaServer Pages (JSPs), 
which are web pages that contain HTML and embedded 
programming code that is executed by a server. When a web 
browser makes a request from the web application 42 for a 
web page, the server executes the embedded program, and 
the HTML provides the page layout that will be returned to 
the web browser. The programming code provides the 
processing for the page, such as delivering search data 
entered on a web page to the database for lookup. It would 
also format the results of that search as HTML and send it 
back to the client device 12 for display. 

[0034] Once communication between the client device 12 
and the image gateway 18 has been established, the client 
device displays a web page from the gateway server 20 
indicating what web applications 42 are available to user. In 
a preferred environment, the available web applications 42 
arc displayed via hyperlinks. For example, the web page 
displayed to the user may display links such as "Upload 
Images," and "Search For Images," which link to corre- 
sponding web applications 42. 

[0035] In response to the user selecting a web application, 
the gateway server 20 connects the client device with the 
selected web application 42. Those with ordinary skill in the 
art will appreciate that the connection is preferably estab- 
lished with a secure handshake mechanism. 



[0036] As noted above, most of the web applications 42 
deal with capturing images and manipulating metadata asso- 
ciated with those images. Consider for example a web 
application 42 that assist a user in making a corporate 
directory. When run on a client device 12, such as a digital 
camera, the web application 42 would prompt the camera 
user to take pictures of all the employees in the company. 

[0037] As shown in FIG. 2 showing example metadata 
fields associated with images for the corporate directory 
application, the user would then provide for each photo the 
appropriate information to populate the metadata fields. The 
metadata may include both standard data regarding the 
captured image and specialized data pertaining to the appli- 
cation. For the corporate directory application, the metadata 
pertaining to the directory application might include the 
employee's name, mail stop, hire date, position, phone 
number, and so on. 

[0038] Requiring that the user manually enter all of the 
employee information for each employee would be tedious 
and time consuming. A further aspect of the present inven- 
tion provides a client -server approach for merging metadata 
with the images. 

[0039] FIG. 3 is a flow chart illustrating a process for 
merging metadata with digital images in accordance with 
one preferred embodiment of the present invention. The 
process begins by allowing a user to upload data to the 
gateway server 20 in step 100, and associating the data with 
an account in step 102. The user may upload the data from 
a conventional computer, the client device, or any other 
source, and the account may represent a user account or a 
corporate account. 

[0040] FIG. 4 is a diagram showing the data that a user 
may upload in the corporate directory example. Assume that 
a user, John Smith, is responsible for creating the corporate 
directory using the corporate directory web application. The 
data that John would upload to the gateway server 20 may 
be an employee database 202 listing all employees and their 
employee information. Once uploaded, the employee data- 
base 202 would be stored and associated with John Smith's 
account 200. 

[0041] Referring again to FIG. 3, other sets of data may 
also be optionally uploaded to the gateway server 20 and 
associated with the account in step 104. FIG. 5 is a block 
diagram illustrating two additional data sets 204 and 206 
that have been uploaded and associated with John Smith's 
account 200. 

[0042] Referring again to FIG. 3, the uploaded data that is 
associated with the account may be further associated with 
one or more web applications 42 in step 106. FIG. 6 is a 
block diagram illustrating that the Employee database 202 
that is associated with John Smith's account 200 could be 
further associated with a "Corporate Directory" web appli- 
cation 208. 

[0043] Referring again to FIG. 3, the gateway server 20 
provides the web application 42 to the client device when 
requested and transmits the previously uploaded data asso- 
ciated with the web application 42 in step 108. The previ- 
ously uploaded data is sent to the device in a format 
compatible with the display characteristics of the client 
device 12. In one embodiment, the uploaded data could 
specified using XML and the server would apply XSL 
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stylesheets depending on the device type. The server could 
have a standard set of stylesheets to apply or custom 
stylesheets could be provided when the data is initially 
uploaded to the server. When an image requires the input of 
metadata, the data transmitted to client device 12 is dis- 
played for selection by the user in step 109. In response to 
the user's selection, the web application 42 uses the selected 
data to automatically populate the metadata fields associated 
with the image file in step 110. 

[0044] In the corporate directory application example, the 
user would choose to open either the employee database 202 
or the data sets 204 and 206. If the user selects the employee 
database 202, then a list of employee names would be 
displayed, allowing the user to select the appropriate 
employee's name. The employee information for the 
employee would then be used to populate the corresponding 
metadata fields in the employee's image file. 

[0045] If the user does not find the desired data displayed 
on the client device 12, the user manually enters the data to 
populate the metadata fields in step 112. Once this data is 
entered into the client device 12, it is then transmitted to the 
gateway server 20 in step 114. As shown in FIG. 7, the 
gateway server 20 may store this data in a history list 210 
within the user's account and associate the history list 210 
with the web application 42 from which it was entered in 
step 116. The history list may also be associated with a 
specific metadata field within the web application 42. Refer- 
ring again to FIG. 3, when the user accesses this web 
application 42 in the future, the gateway server 20 will 
transmit the data in the history list to the web application 42 
in step 118, thus saving the user from having to manually 
enter the data again. 

[0046] According to the present invention, the need for 
cumbersome data entry on the client device 12 is signifi- 
cantly minimized since all the data needed for the image is 
stored on the gateway sever 20, associated with the user's 
account, associated with a specific web application, and then 
transmitted to the camera when the data is needed. 

[0047] According to yet a further aspect of the present 
invention, XML and XSL(i.e., XML stylesheets) are used to 
enable a user to attach metadata to any image and associate 
one or more XSL stylesheets to the metadata. For example, 
an insurance agent could take a picture or an auto accident 
and submit it to the gateway server 20 with metadata that is 
specific to his/her own company's use of that data. When the 
image and/or its metadata are accessed, the gateway server 
20 uses the stylesheets associated with metadata along with 
information about the presentation characteristics of the 
client device 12 to present the metadata in a manner that is 
consistent with the semantics and meaning of the metadata. 
That is, the gateway server 20 could be a form or report 
depending on the request from the user. 

[0048] The gateway server 20 would not need to under- 
stand the meaning or the format of the metadata. In fact, 
each image could have its own unique set of XML tags that 
are used to specify its metadata. Each image in this case 
would have its own stylesheet or set of stylesheets that the 
gateway server 20 uses to customize the display of the 
metadata based on the display device and the request. In a 
preferred embodiment, however, standard sets of XML tags 
are provided for use with images depending on the web 
application 42 that the gateway server 20 supports, instead 
of defining different XML tags for each image. 



[0049] Although the present invention has been described 
in accordance with the embodiments shown, one of ordinary 
skill in the art will readily recognize that there could be 
variations to the embodiments and those variations would be 
within the spirit and scope of the present invention. Accord- 
ingly, many modifications may be made by one of ordinary 
skill in the art without departing from the spirit and scope of 
the appended claims. 

What is claimed is: 

1 A method for merging metadata with digital images, 
wherein the digital images are stored as image files having 
metadata fields associated therewith, the method comprising 
the steps of: 

(a) receiving data uploaded from a remote location; 

(b) providing a web application to a client device over a 
network when requested, and transmitting the previ- 
ously uploaded data to the web application; 

(c) when an image requires input of metadata, displaying 
the uploaded data on the client device for selection by 
the user; and 

(d) in response to the user's selection, using the selected 
data to automatically populate the metadata fields asso- 
ciated with the image file. 

2 The method of claim 1 further including the step of 
associating the uploaded data with an account. 

3 The method of claim 2 further including the step of 
associating the uploaded data with the web application. 

4 The method of claim 3 further including the steps of 
receiving the uploaded data from the client device. 

5 The method of claim 4 further including the step of 
receiving another data set and associating the data set with 
the account. 

6 The method of claim 4 further including the steps of: 

allowing the user to manually enter data on the client 
device; 

receiving the manually entered data transmitted from the 
client device; 

associating the data with the web application from which 
it was entered; and 

when the user subsequently accesses this web application, 
transmitting the manually entered data to the web 
application for selection by the user, thereby saving the 
user from having to manually enter the data again 

7 The method of claim 6 further including the step of 
storing the manually data in a history list within a user 
account. 

8 The method of claim 7 further including the step of 
associating the history list with a specific metadata field 
within the web application. 

9 The method of claim 4 further including the step of 
using XML tags to allow metadata fields to be defined for 
images and associating one or more XSL stylesheets to the 
metadata. 

10 The method of claim 9 further including the step of 
using the stylesheets associated with metadata and presen- 
tation characteristics of the client device to present the 
metadata in a manner that is consistent with the semantics 
and meaning of the metadata. 
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11 A system for merging metadata with digital images, 
wherein the digital images are stored as image files having 
metadata fields associated therewith, the method comprising 
the steps of: 

means for receiving data uploaded from a remote loca- 
tion; 

means for providing a web application to a client device 
over a network when requested, and transmitting the 
previously uploaded data to the web application; 

means for when an image requires input of metadata, 
displaying the uploaded data on the client device for 
selection by the user; and 

means for in response to the user's selection, using the 
selected data to automatically populate the metadata 
fields associated with the image file. 

12 The system of claim 11 wherein the uploaded data is 
associated with an account. 

13 The system of claim 12 wherein the uploaded data is 
associated with the web application. 

14 The system of claim 13 wherein the uploaded data is 
received from the client device. 

15 The system of claim 14 wherein another data set is 
received and associated with the account. 

16 The system of claim 14 further including: 

means for allowing the user to manually enter data on the 
client device; 

means for receiving the manually entered data transmitted 
from the the client device; 

means for associating the data with the web application 
from which it was entered; and 

means for responsive to the user subsequently accesses 
this web application for transmitting the manually 
entered data to the web application for selection by the 
user, thereby saving the user from having to manually 
enter the data again. 

17 The system of claim 16 wherein the manually data is 
stored in a history list within a user account. 

18 The system of claim 15 wherein the history list is 
associated with a specific metadata field within the web 
application. 

19 The system of claim 14 wherein XML tags are used to 
allow metadata fields to be defined for images and wherein 
one or more XSL stylesheets are associated with the meta- 
data. 



20 The system of claim 19 wherein the stylesheets asso- 
ciated with metadata are used along with the presentation 
characteristics of the client device to present the metadata in 
a manner that is consistent with the semantics and meaning 
of the metadata. 

21 An online system comprising: 

a client device having files stored thereon, wherein each 
of the files has metadata fields associated therewith; 

a server in communication with the client device over a 
network, the server for supplying a web application to 
the client device and further for storing previously 
uploaded data associated with an account; and 

a browser running on the client device for interacting with 
the web application, such that when the web applica- 
tion identifies the account to the server, the server 
transmits the uploaded data to the web application so 
that when a file requires the input of metadata, the 
uploaded data may be displayed on the client device for 
selection by the user, wherein in response to the user's 
selection, the web application uses the selected data to 
automatically populate the metadata fields associated 
with the file. 

22 The online system of claim 21 wherein the files 
comprise image files. 

23 The online system of claim 22 wherein the web 
application performs imaging related function on the image 
files. 

24 The online system of claim 23 wherein the client 
device comprises an image capture device. 

25 The online system of claim 24 including a plurality of 
client devices that communicate data in different formats, 
and a plurality of online photo service sites, wherein each of 
the photo service sites utilize different data models, the 
server further including; 

a meta-application for defining a common data model 
format for the different formats of the photo service 
sites, such that when a request is received from a client 
device for photo services from a particular photo ser- 
vice site, the request is passed to the photo service site, 
and wherein when a response from the photo service 
site is received, the response is converted from the data 
model of the photo service site to the common data 
model format, the converted request is then presented 
to the requesting client device in the data format 
required by the requesting client device. 

* * * * * 
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